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ABSTRACT 

The National Household Education Survey (NHES) is a 
data collection system of the National Center for Education 
Statistics which has as its mission the collection and publication of 
data on the condition of education in the United States. The NHES 
provides descriptive data through a random digit dialed telephone 
survey of the nonins t i tut ional ized civilian population of the United 
States. Approximately 60,000 households are screened for each 
adminis trat ion, and individuals who meet the study criteria are 
sampled for more detailed or extended interviews. Data quality is a 
central issue for the NHES. Cognitive laboratory research, one of the 
many efforts to reduce nonsampling error in surveys, draws on the 
theories and methods of cognitive psychology. Concurrent and delayed 
methods are used in cognitive laboratory research procedures that can 
include individual interviews and focus groups. Cognitive laboratory 
activities are typically audiotaped or videotaped to provide a record 
of the research. Several examples of the use of cognitive laboratory 
research in the conduct of the NHES are given. These examples 
illustrate how taping and analysis of live interviews measured 
interviewer and respondent behavior during data collection and 
contributed to understanding words and items that could cause 
difficulty. Three appendixes present a moderator's topic guide for 
parent groups, a similar guide for adolescent groups, and a form for 
coding int ervi ewer-respondent int erac t i ons . (Contains 1 appendix 
table and 23 references.) (SLD) 
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Background 

The National Household Education Survey 
(NHES) is a data collection system of the 
National Center for Education Statistics (NCES), 
which has as its legislative mission the collection 
and publication of data on the condition of 
education in the Nation. The NHES is 
specifically designed to support this mission by 
providing information on those educational issues 
that are best addressed by contacting households 
rather than schools or other educational 
institutions. The NHES provides descriptive data 
on the educational activities of the U.S. 
population and offers policymakers, researchers, 
and educators a variety of statistics on the 
condition of education in the United States. 

The NHES is a telephone survey of the 
noninstitutionalized civilian population of the 
United States. Households are selected for the 
survey using random digit dialing (RDD) 
methods, and data are collected using computer- 
assisted telephone interviewing (CATI) 
procedures. Approximately 60,000 households 
are screened for each administration, and 
individuals within households who meet 
predetermined criteria are sampled for more 
detailed or extended interviews. The data are 
weighted to permit estimates of the entire 
population. The NHES survey for a given year 
typically consists of a Screener, which collects 
household composition and demographic data, 
and extended interviews on two substantive 
components addressing education-related topics. 

The primary purpose of the NHES is to conduct 
repeated measurements of the same phenomena 
at different points in time, although one-time 
surveys on topics of interest to the Department 
of Education may also be conducted. 
Throughout its history, the NHES has collected 
data in ways that permit estimates to be tracked 
across time. This includes repeating topical 
components on a rotating basis in order to 



provide comparative data across survey years. In 
addition, each administration of the NHES has 
benefited from experiences with previous cycles, 
resulting in enhancements to the survey procedures 
and content. Thus, while the survey affords the 
opportunity for tracking phenomena across time, it is 
also dynamic in addressing new issues and including 
conceptual and methodological refinements. 

A new design feature of the NHES program 
implemented in the NHES:96 is the collection of 
demographic and educational information on 
members of all screened households, rather than just 
those households potentially eligible for a topical 
component. In addition, this expanded screening 
feature includes a brief set of questions on an issue 
of interest to education program administrators or 
policymakers. The total Screener sample size is 
sufficient to produce state estimates of household 
characteristics for the NHES:96. 

Full-scale implementations of the NHES have been 
conducted in 1991, 1993, 1995, and 1996. Topics 
addressed by the NHES:91 were early childhood 
education and adult education. The NHES:93 
collected information about school readiness and 
school safety and discipline. The 1991 components 
were repeated for the NHES:95, addressing early 
childhood program participation and adult education. 
Both components underwent substantial redesign to 
incorporate new issues and develop new 
measurement approaches. In the NHES:96, the 
topical components are parent/family involvement in 
education and civic involvement. The NHES:96 
expanded screening feature includes a set of 
questions on public library use. 

In addition to its topical components, the NHES 
system has also included a number of methodological 
investigations. These have resulted in technical 
reports and working papers covering diverse topics 
such as telephone undercoverage bias, proxy 
reporting, and sampling methods. This series of 
technical reports and working papers, which includes 
this report, provides valuable information on ways of 
improving the NHES. 



This report describes cognitive laboratory 
research and analysis of taped interviews in the 
NHES. The description of these data quality 
activities is illustrated with selected examples 
from survey components in theNHES:91 and the 
NHES:93. The report is intended to give an 
overview of these methods as they are useid in a 
major education data collection, not to describe 
the results of the activities in detail. 

Overview of Data Quality Activities in the 
NHES 

Many social scientists rely on data from large- 
scale surveys to conduct their research, and it is 
assumed that the data are accurate and reliable. 
However, the quality of survey data is affected 
by both sampling and nonsampling error. 
Substantial research has been conducted to 
measure and to find ways of reducing sampling 
error, error that results from surveying only a 
portion of the inference population. Other 
research has been conducted to find ways of 
reducing nonsampling error, error that arises 
from behavior of the respondent or the 
interviewer or both. In order to reduce this type 
of error, researchers expend considerable effort 
to 1) design questions that are clear and easily 
comprehensible to the respondent, 2) reduce 
nonresponse among sampled persons, and 3) 
train interviewers to ensure consistency in the 
way that questions are asked (Groves 1989, 1-6). 

Data quality is a central issue for the NHES, and 
care is taken at each stage of the survey design 
and data collection process to secure high quality 
data. Several methods have been used in the 
NHES to reduce nonsampling error by increasing 
the comprehensibility of survey questions and 
maximizing interviewer consistency. Each 
method elicits different information and suggests 
specific means of improving the survey (Presser 
and Blair 1994). For instance, for some survey 
topics, cognitive research explores the salience of 
those topics with participants drawn from the 
intended population of interest. Then, drafts 



of NHES questionnaire items are presented for 
review and comment to Technical Review Panels 
composed of both methodological experts and 
experts in the content areas from which the survey 
topics have been drawn. Cognitive research 
consisting of intensive interviews and focus groups 
is conducted after survey items are developed, and 
changes are made to the instruments based on the 
findings from this research. Following the 

conclusion of cognitive laboratory research, the 
instruments are programmed for computer 

administration, and one or more pretests are 
conducted^ using live, computer-assisted telephone 
interviews "(CATI). These efforts result in 
adjustments to the interview instruments. Pretest 
results come from both the actual responses and from 
what is learned about respondent comprehension and 
respondent-interviewer interaction through extensive 
monitoring by project staff and in-depth debriefings 
of interviewers. 

During data collection, interviews are regularly 
monitored by project staff and Telephone Research 
Center staff, and interviewers receive quick and 
specific feedback on their performance. In addition, 
a small number of interviews have been recorded, 
and staff have analyzed the observed interviewer- 
behavior interaction. These activities are used in 
conjunction with the results of a reinterview of a 
subsample of the respondents and a comparison of 
the NHES findings with other surveys to investigate 
and offer probable explanations for any unexpected 
results and to improve future administrations of the 
survey (Brick and West 1992; Brick, Rizzo, and 
Wemimont forthcoming; Brick, Collins, Nolin, and 
Davies forthcoming). 

Two data quality activities of the NHES, cognitive 
laboratory research and analysis of recorded 
interviews, are the focus of this report. Cognitive 
laboratory research plays a key role during the 
design of each NHES survey. This research utilizes 
the methods of cognitive psychology to increase 
understanding of the ways that respondents 
comprehend survey instructions and questions, recall 
requested information, and respond to the influence 
of word and question order. It has been used in the 
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NHES to explore the feasibility of potential 
survey topics, to assess the reliability of classes 
of respondents, and to test questionnaires in 
order to reduce measurement error. Maximizing 
the clarity and conciseness of the questions is a 
particular concern for a survey like the NHES, in 
which a random sample of persons is surveyed 
without any previous introduction to the study. 
Respondents are typically unfamiliar with 
surveys, some have relatively low educational 
attainment, and all possess perceptions unique to 
their own social experience. An understanding 
of respondents’ general perceptions of the survey 
topics and the salience of the issues being 
explored, as well as knowledge about the 
strategies that respondents are likely to use to 
answer survey questions, has the potential of 
reducing measurement error. So also does 
creating an interview context in which 
respondents are likely to respond accurately in 
terms of their own experience (Schwartz 1995). 

The analysis of recorded interviews conducted 
during the NHES:91 and the NHES.93 has 
provided a useful assessment of data quality. 
During data collection for those surveys, a 
limited number of interviews were tape-recorded 
and later analyzed to systematically assess the 
interview process and to evaluate respondent 
comprehension during the interview. A different 
method of analysis was used each year. For the 
NHES:91, tapes were analyzed for content; for 
the NHES.93, each interviewer-respondent 
interaction was coded. 

Following a brief account of the emergence of 
cognitive laboratory research as a tool for survey 
researchers and a description of cognitive 
laboratory methods and procedures, the 
application of these procedures in the NHES:91 
and NHES: 93 is described. An account of the 
assessment of interviews recorded during data 
collection in the NHES:91 and NHES:93 is then 
presented. 



Overview of Cognitive Laboratory 
Research 

Cognitive laboratory research, one of the many 
efforts to reduce nonsampling error in surveys, draws 
on the theories and methods of cognitive psychology. 
Collaboration between survey researchers and 
cognitive psychologists began in 1978. To a great 
extent this collaboration was sponsored and funded 
by government agencies in the United States, the 
United Kingdom, and West Germany in order to 
improve the quality of the survey data that served as 
the basis for policy decisions. Cognitive 
psychologists were motivated to join in the 
interdisciplinary effort because they were provided 
with the opportunity to study cognitive processes in 
a natural setting rather than in the artificial setting of 
the laboratory (Jobe and Mingay 1991). Examples 
of this research (as cited by Jobe and Mingay) 
include research on autobiographical memory, 
cognitive interviewing techniques, experiential 
memory, the influence of response alternatives on 
responses, memory for dates and events, question 
comprehension, question ordering and wording, and 
response order. Most of the research was published 
in the 1980s. According to Lessler, Tourangeau, and 
Salter (1989), despite government funding of early 
cognitive research, cognitive laboratory procedures 
were underutilized in survey research for Federal 
agencies. Their own program for investigating the 
cognitive aspects of survey methodology at the 
National Center for Health Statistics, described in 
Lessler, Bercini, et al. (1986) and Lessler, 
Tourangeau, and Salter (1989), was an exception. In 
this decade, however, more Federal studies have 
incorporated cognitive laboratory research (see, for 
example. Bates and DeMaio 1990; Campenelli et al. 
1991; Cantor and Edwards 1992; Cantor et al., 
1995; Dippo and Norwood 1992; Jenkins, Ciochetto, 
and Davis 1992; Jenkins 1992). Cognitive laboratoiy 
research in the NHES has been part of this effort. 

In the next sections, cognitive laboratory methods are 
discussed first, followed by a description of the 
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procedures that are used in combination with 
those methods, and then by a brief account of 
how data from cognitive laboratory research are 
analyzed. 

Cognitive Laboratory Methods 

Forsythe and Lessler (1991) provide a valuable 
classification of cognitive laboratory methods 
from which this summary is drawn. They 
differentiate the methods used in cognitive 
laboratory research along two dimensions: the 
timing of the cognitive tasks, and the level to 
which the respondent must attend to his or her 
cognitive processes in order to provide the 
information requested. Some methods are called 
concurrent methods because the cognitive tasks 
of providing information and articulating 
meaning and/or recall processes are performed at 
the same time. The respondent’s attention is 
specifically directed to the response process at 
the same time he or she is required to retrieve 
and provide information. The respondent must 
not only respond to the question, but also talk 
about the mental steps involved in retrieving and 
sorting the requested information. Some 
cognitive laboratory methods are termed delayed 
because they involve cognitive tasks that the 
respondent performs after the process of 
retrieving and providing information. That is, 
the respondent first provides the information and 
then, shortly afterwards or when a somewhat 
longer period of time has elapsed, the respondent 
reports on his or her cognitive processes. 
Selection of a concurrent or delayed method is 
guided by the specific goals of the research. 

Two concurrent methods, concurrent think- 
alouds and concurrent probing, are frequently 
used in cognitive laboratory research. The 
concurrent think-aloud method requires the 
respondent to verbalize his or her thoughts 
spontaneously while answering the interview 
questions. This method is based on protocol 
analysis (see Ericsson and Simon 1980 and 
Ericsson and Irwin 1984) in which participants 
are asked to think aloud as they solve a problem. 
When this approach is used in cognitive 



laboratories to test a questionnaire, respondents are 
asked to think aloud as they answer each question in 
order to reveal the strategies they use to recall the 
information. They talk through the steps they take 
to produce an answer while they are producing the 
answer. For example, when asked a question that 
requires recall over an extended period, a respondent 
might think in terms of months or seasons, and that 
would become apparent as he or she spoke while 
retrieving the information. (For example, a 
respondent might say, "Let’s see, in June we went to 
the beach, and I wasn’t taking courses then. I must 
have started in July.") This activity is not an easy 
one for many respondents because the thoughts that 
they are asked to verbalize are usually not expressed 
aloud. Typically, we do not think about "how we 
think" when generating responses to questions, and 
some respondents do not articulate their thoughts 
easily, even when they are attending to them. A 
variant on the think-aloud method, in which the 
researcher takes a more directive role, is the 
concurrent probe. Immediately following a 
response, the participant is asked about some aspect 
of the question or response in order to elicit specific 
and detailed information on response strategy. 

Cognitive laboratories also use delayed, or 
retrospective, methods. Delayed methods tap 
information about respondent knowledge, 
comprehension, and recall, as well as general 
perceptions about survey items that may affect 
motivation, particularly those that are formed over 
time (Forsythe and Lessler 1991). These methods 
call for the administration of the survey under 
conditions similar to those encountered during the 
actual data collection. Then, after all questions are 
answered, information about the survey and the 
response process is elicited. The researcher may use 
either nondirective or directive retrospective probes 
to draw out this information. For example, a 
researcher may refer to a particular answer and ask 
the respondent to think-aloud retrospectively, 
explain what he or she was thinking when 
responding. Or, in a more directive probe, the 
researcher may refer to a particular term that the 
respondent used and ask for the respondent’s 
definition of that term in more detail. The elapsed 
time between administration of the interview and the 
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procedures to measure cognitive processes varies 
from immediately following administration ofthe 
survey to a few days. In the former instance, 
information requested of the respondent resides 
in short-term memory and is generally easily 
retrievable. In the latter case, the time interval 
requires the respondent to reconstruct what he or 
she was thinking at the time the question was 
asked. It also permits access to opinions 
regarding the survey that have been formed over 
the time between asking the question and 
probing for more information. 

Both concurrent and delayed methods are used in 
cognitive laboratory research for the NHES. 
Concurrent methods are most often employed in 
testing questionnaire items. Delayed methods 
are most commonly used in the early stages of 
the design of an NHES survey to explore 
research topics and determine the appropriate 
respondents for those topics. Delayed methods 
may also be used to test questionnaire items. 

Cognitive Laboratory Procedures 

Concurrent and delayed methods can be 
employed with different cognitive laboratory 
procedures, creating an array of tools for the 
researcher (table 1). The procedures commonly 
used are individual, in-depth interviews and 
focus groups. In the NHES, focus groups have 
been used during the earliest stages of survey 
design to assess the feasibility of potential 



research topics and the suitability of a class or 
classes of respondents, given the topic. In-depth 
face-to-face interviews and focus groups have been 
used to evaluate the clarity and efficacy of 
questionnaire items and to improve and test versions 
of the instruments. The following sections describe 
interviews and focus groups and the specific 
cognitive laboratory methods typically used with 
these procedures. 

Individual Interviews 

Individual interviews are an important part of 
cognitive laboratory research. During cognitive 
interviews, the researcher is able to interact with one 
participant at a time, and give his or her full 
attention to the information offered by the 
participant. This procedure affords the researcher the 
flexibility to adapt the methodology to the participant 
and to pursue valuable and perhaps unanticipated 
lines of inquiry. Both concurrent and delayed 
cognitive laboratory methods can be used with 
individual interviews. 

Cognitive laboratory activities to test NHES survey 
items usually include concurrent think-aloud 
interviews in which respondents verbalize their 
response strategies as they answer each question. 
The absence of intervention by the interviewer 
permits unexpected information to emerge. 
However, the concurrent think-aloud method can 
influence the responses that are obtained. Because 
the respondent is attending to the process of 
retrieving information and deciding upon response 



Table 1 .— Cognitive laboratory techniques used in the NHES 



Procedures 


Methods 


Concurrent 


Delayed 


Individual interviews 


Concurrent think-aloud 


Retrospective think-aloud 




Concurrent probing 


Retrospective probing 


Group discussion 




Focus group 
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strategies, he or she may be more careful in 
searching for accurate information than a survey 
respondent under typical interviewing conditions. 

Although respondents are instructed, in the think- 
aloud method prior to administration of the 
questionnaire, the procedure is awkward for 
some respondents because they must consider 
both the response and the strategy for obtaining 
the response. Some respondents are simply 
unable to report on their response strategies 
while answering questions. In addition, because 
some respondents focus intently on generalizing 
an out-loud response to a question, they may 
have difficulty focusing on the subsequent 
questions, which can reduce the accuracy of their 
self-reporting. 

In these cases, another concurrent method, the 
concurrent probe, may be used. During the 
interview, the respondent answers questions 
without having to attend to retrieval processes at 
the same time. However, just after a response is 
given, the interviewer probes to ascertain details 
of the response process including relevant 
information about knowledge and motivation to 
respond. 

Delayed cognitive methods may also be used in 
conjunction with individual interviews. In these 
interviews, after the respondent has answered all 
questions, he or she is debriefed about his or her 
responses. Usually certain questions or groups 
of questions are targeted for debriefing. Delayed 
methods used with interviews can be 
nondirective, as in the retrospective think-aloud 
technique in which the respondent is simply 
asked to recall and state what were his or her 
thoughts when responding to particular questions. 
Retrospective probing . is more structured; 
information about specific aspects of content and 
time domains is elicited by the researcher. 
Questions about the meaning of terms, the 
techniques used to recall information, or 
reference periods for time-related questions 
might be asked. The respondent’s assessment of 
the accuracy of the original responses can also 
be obtained. 



Focus Groups 

Focus groups are a cognitive laboratory method that 
can generate valuable insight on respondent 
comprehension, recall, and motivation to answer 
survey items. In the NHES, they are a means of 
exploring potential survey topics, establishing the 
appropriate respondents to the surveys, and 
evaluating drafts of interviews. First used in social 
science research over 30 years ago, focus groups are 
small groups of 6 to 10 individuals who are brought 
together to react to a stimulus situation or discuss a 
particular issue or set of issues (see Merton, Fiske, 
and Kendall 1956 for a discussion of the early use of 
focus groups). Focus group participants are selected 
according to criteria determined by the research topic 
and the research goals. Groups are often composed 
of participants who share many sociodemographic 
characteristics because interacting with others who 
are similar can encourage self-disclosure (Merton, 
Fiske, and Kendall 1956, 137-138). Alternatively, 
especially when the number of groups to be 
conducted on a topic is limited, conducting groups 
homogenous on several variables (e.g., older white 
males, younger white males, older black males, older 
white females, etc.) may not be feasible. In such 
instances, diversity on sociodemographic variables is 
accepted, and homogeneity with regard to the 
particular research topic can be sought. For 
example, for a study in which it is anticipated that 
experience will vary by the grade of the student, one 
might want to include youth from a narrow grade 
range but who attend public and private schools, who 
are black, white, and Hispanic, and who live in 
urban and suburban areas. 

Protocols designed with the stage of development of 
the survey and the particular research purpose in 
mind are created to guide focus group discussions. 
They consist of unstructured, semistructured, or 
structured questions and probes. The intent is to 
capture a range of responses from participants, to 
elicit information that is as specific as possible, to 
encourage participants to contribute perceptions that 
are both cognitively and emotionally meaningful, and 
to learn about the personal context from which their 
responses are drawn (Merton, Fiske, and Kendall 
1956). For example, protocols for focus groups 
conducted for exploratory research will contain open- 
ended questions to encourage the respondents to 
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voice their own opinions and perceptions as 
unconstrained as possible by the 
conceptualizations of the researchers. Protocols 
for focus groups conducted for the purpose of 
testing survey instruments may use more 
directive techniques specified by semistructured 
or structured questions. Typically, participants 
in these focus groups are prepared for the 
discussion by having the survey administered to 
them beforehand. The protocol is constructed to 
decompose responses and explore comprehension 
and time references. The focus group discussion 
encompasses thoughts participants have had in 
the period intervening between the interview and 
discussion, and elicits attitudes about the survey 
that may have developed over time. 

Focus groups are led by trained moderators who 
direct the discussion but allow the respondents to 
voice their own opinions and perceptions about 
the survey. This method affords participants the 
benefit of group interaction. In exploratory 
research, group discussion can yield constructive 
information about respondents’ general 
knowledge of survey topics and related issues 
and can help researchers to use familiar terms 
when designing questionnaire items. In groups 
conducted to test questionnaires, comments by 
some group members may help others recall 
thoughts and perceptions that occurred at the 
time they responded to the survey or encourage 
them to offer opinions about the survey. Because 
several respondents take part in the discussion 
simultaneously, focus groups offer researchers a 
labor- and time-efficient method of gathering 
qualitative data to improve survey content and 
reduce measurement error. 

Focus groups have some limitations, however. 
Because several participants are involved in the 
task, the researcher must pay close attention to 
group process as well as to the research 
objective. Some participants may attempt to 
dominate the group, subvert the task, fulfill 
personal goals, or otherwise adhere to their own 
agenda. Alternatively, other participants may be 



swayed by group opinion or may be reluctant to 
divulge information in front of others that they 
would reveal to a researcher in an individual 
interview. Focus group participants must be assured 
of the confidentiality of their opinions, which are to 
be used for research purposes only. Focus group 
data, whether in the form of notes or video- or 
audiotapes of the proceedings, must be treated with 
extreme care by researchers to protect that 
confidentiality. Also, focus groups, like the other 
methods discussed here, do not yield findings that 
are generalizable to a larger population, so their 
results must always be interpreted with caution. 

Analysis off Cognitive Laboratory Data 

Cognitive laboratory activities are typically 
audiotaped or videotaped to provide a record of the 
research. Findings are drawn from content analysis 
of the tapes, or, in some instances, of transcriptions 
of the tapes. However, analysis of transcriptions 
cannot take the place of listening to or viewing the 
tapes or observing the sessions because the context 
of a person’s remarks (tone of voice, posture, etc.) is 
also useful information. In addition to the tapes, 
data may be generated by notes taken by an observer 
during a focus group and notes recorded by the 
interviewer or focus group moderator immediately 
following the activity. 

The specific analytical procedure used by the 
researcher depends upon the research goal. Data 
may be precisely coded, and words and phrases 
compared and contrasted in research conducted to 
generate conceptual categories, or the data may be 
subjected to a more general analytical search for 
major points. Data collected to assess 
comprehension, recall, and the sensitivity of 
questionnaire items can be examined for the range of 
answers, specificity of ideas, and patterns of response 
(Krueger 1988, 106-121). Although findings are 
often based upon responses that emerge repeatedly 
throughout the research, valuable insights can be 
generated by one comment made by a single research 
participant. Therefore, a search for key comments, 
not only patterns, is part of the analysis of cognitive 
laboratory data. 
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In the NHES, individual interviews are 
audiotaped, and focus groups are both audio- and 
videotaped. An observer also sits behind a one- 
way mirror and takes notes during the focus 
group. The interviewer or focus group 
moderator records his or her notes on the 
research immediately following the activity. 
Answers to the general questions that guide the 
research and are embedded in the protocol are 
drawn from reviewing the tapes and the notes of 
the observer and the interviewer or moderator. 
Recommendations are based on these findings. 

Examples of Cognitive Laboratory 
Research Conducted for the NHES 

Cognitive laboratory research has been conducted 
in the design phase of each NHES data 
collection, and the role of this research in the 
development of each survey has increased over 
time. The examples presented here are not 
intended to be comprehensive with regard to the 
issues explored in the cognitive laboratories, the 
procedures used during the design of each survey 
component, or the findings that emerged from 
the research. However, they illustrate ways that 
cognitive laboratory methods and procedures 
have been used to investigate various issues for 
different topical components with beneficial 
results for the overall data quality of the NHES. 

Exploring the Feasibility of a Research Topic 
and the Appropriateness of Respondents: 
School Safety and Discipline 

Cognitive laboratory research was undertaken to 
examine the feasibility of school safety and 
discipline as a topic for the NHES:93. Interest 
in School Safety and Discipline as a potential 
topic for the NHES intensified when this topic 
became one of the President’s and Governors’ 
National Education Goals. While some data on 
student drug use, victimization, and teachers’ 
perceptions of school safety were published 
(National Education Goals Panel 1991, 1992), 
there was a need for more data, not only on 
student victimization, but also on overall 



perception of the "learning environment" at school. 
In addition, NCES was interested in placing parents’ 
and students’ reports in context by gathering data on 
parent approval or disapproval of alcohol use by 
their children, the extent of parent support of school 
discipline policy, and certain school and 
neighborhood characteristics. Substantive experts 
agreed with the need for more data but thought that 
obstacles to collecting it might exist. There were 
concerns that parents may not pay much attention to 
issues related to safety at school, that students may 
not be reliable respondents because of their age, and 
that concepts used by educational policymakers and 
researchers would not be readily understood by 
parents and students. 

Because of the open-ended format and the potential 
for new information to emerge from group 
interaction, focus groups were chosen as the 
appropriate methodology to explore the salience of 
the issues and parent and student comprehension of 
the concepts related to school safety and discipline. 
Professionals had identified areas related to safety 
and discipline in school for which data were needed. 
Small groups of 8 to 10 parents or students were 
brought together to discuss aspects of the topic and 
to test the ability of potential respondents to provide 
the data. The group format encouraged a range of 
opinions, with the comments of some members of 
the groups serving as stimuli for others. 

Participants. During May and June of 1991, seven 
focus groups were conducted to test the feasibility of 
a School Safety and Discipline topical component for 
the NHES. All of the groups were conducted near 
the Washington, DC or Baltimore, Maryland, 
metropolitan areas. Participants were recruited 
through flyers posted in public places such as office 
buildings and grocery stores, and through an 
advertisement placed in a local newspaper in 
Gaithersburg, Maryland, a suburb approximately 35 
minutes outside of Washington, D.C. Participants 
with experience in inner-city schools were recruited 
through a market research firm in Baltimore, 
Maryland. Volunteers were administered a brief 
screener so that participants could be selected 
according to specific characteristics that the included 
grade of student, type of school, race/ethnicity, and 
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parents’ occupation. Four of the groups were 
composed of parents; one group of parents of 
students in grades 1 through 6, two of parents of 
students in grades 7 through 9, and one of 
parents of students in grades 10 through 12. 
Groups contained both mothers and fathers, but 
not two parents from the same family. In 
addition, three groups of students were 
conducted, two composed of students in grades 
7 through 9, and one of students in grades 10 
through 12. Five of the groups (three parent and 
two student groups) were conducted in 
Gaithersburg, and two (one parent and one 
student group) were conducted in Baltimore. 
The prior written consent of a parent was 
obtained for each student who participated in a 
focus group, and all participants, parents and 
students, gave written consent before the 
discussion began. Each of the participants was 
paid an honorarium. 

The groups were racially and ethnically diverse, 
and respondents were from different 
socioeconomic levels as measured by occupation. 
Twenty-five of the parent participants and 1 8 of 
the youth participants were white, and 10 of the 
parents and 6 of the youth were black or 
Hispanic. Approximately half of the parent 
participants were in relatively low-wage 
occupations that did not require postsecondary 
education, for example, automobile salesman or 
maintenance worker. About half of the student 
participants came from households in which the 
highest occupation was also a relatively low- 
wage one. Mothers and fathers (from different 
families) participated, as did male and female 
students. 

Protocols. Moderator’s topic guides were 
developed to guide the focus group discussions 
(see Appendix A and Appendix B). They were 
organized around four broad topics: safety in the 
schools, alcohol and other drug use, alcohol and 
drug education, and school discipline policy. 

The protocols contained both structured and 
unstructured questions and probes designed to 
elicit in-depth and personal discussion of each of 
the topics. 



Findings. The focus groups demonstrated that 
school safety and discipline was a feasible topic for 
the NHES — one of great interest to both parents and 
students. Most parents held understandings of the 
key concepts consistent with those of the 
professionals who had been consulted in the earliest 
stage of the design. There were few differences 
between the suburban and urban groups with regard 
to level of awareness and concern about unsafe 
incidents at school, although urban parents and 
students regarded lack of safety at school as more 
prevalent and inevitable than did the others. 

One of. the points that the focus group research 
addressed was who would be appropriate respondents 
for this topical component. It was believed that 
safety and discipline could be of concern to parents 
and students at lower as well as higher grade levels; 
however, it had not been determined at what grade 
level parents began to focus on those issues. As 
intended, during the course of the focus group 
discussion, the meanings that the participants 
attached to safety and discipline in school emerged 
relatively unconstrained by the perceptions of 
researchers. The focus group composed of parents 
of elementary school students revealed a range of 
concern about safety issues linked to the age of the 
child. Parents of younger elementary school 
students, children in grades 1 through 3, tended to 
think about safety in terms of freedom from outside 
intruders who could gain access to the school and 
potentially harm their children. These parents were 
less concerned with serious disruption in the 
classroom, fights between students, and weapons and 
drugs in school. Parents with older elementary 
school children begin to anticipate the occurrence of 
unsafe conditions in schools and serious lack of 
discipline in the classrooms. As children entered the 
4th, 5th, and 6th grades, the issues related to 
discipline, fighting, and even weapons in school 
become more salient for parents. Parents whose 
children were to enter middle school in 6th grade 
expressed more anxiety than parents of children who 
would remain in an elementary school through 8th 
grade, and parents of students in junior high (7th-9th 
grade) and high school (9th - 12th grade) saw the 
safety of their children from harm or threats of harm 
by other students as being of central concern. 
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The appropriate parent respondents for this 
survey would be those who were immediately 
concerned about safety and discipline at school 
as it was identified in the National Goals. 
Discussions in the focus groups established that 
parents of students in grades 6 through 12 were 
those parents; they would be the best respondents 
for a broad range of questions on safety and on 
such issues as drug availability and use. 
However, the emerging concerns of parents of 
younger students argued for including them in 
the survey population also. Their comments 
showed that they were beginning to look ahead 
to their children’s middle or junior high school 
years. The focus group findings led to the 
decision to include parents of students in grades 
3 through 5 as respondents for this survey 
component. Based on the findings that some 
subjects (e.g., alcohol and drugs or weapons at 
school) were not salient for this population, it 
was decided to administer to them a subset of 
questions focused on the safety issues most 
relevant to the parents of younger students, for 
example, classroom disruption and bullying. 

It was hypothesized that parents do not 
necessarily have full information from their 
children or from other sources about disruption 
and violence in school, the availability of alcohol 
or other drugs, the school discipline policy, or 
the alcohol/other drug education at their child’s 
school, and the focus group discussions 
supported this conjecture. Some parents reported 
consulting with a child after volunteering for the 
cognitive laboratory research, and thereby 
learning that some of their ideas about safety at 
school were inaccurate. Data from the students 
themselves would be potentially more accurate 
on some subjects; however, there was concern 
about the reliability of students, particularly 
younger students, as respondents. 

Focus groups with students tested their ability to 
report on issues related to school safety and 
discipline. Students in high school were, as 



anticipated, knowledgeable about the full range of 
issues related to school safety and discipline, 
including alcohol and drugs and weapons at school. 
The focus groups also revealed that students in 
grades 7 and 8 were knowledgeable, and at least as 
concerned with their personal safety as were older 
students. In fact, the discussions with these students 
highlighted the problem of older students bullying 
younger students, a topic that was explicitly included 
in the survey because of this finding. None of the 
students who participated in the focus groups had 
difficulty understanding the concepts and terms 
linked to the topics of interest related to school 
safety. Although this round of cognitive laboratory 
research did not include 6th grade students, the 
retrospective reports of the 7th and 8th graders about 
their experiences as 6th graders and their apparent 
reliability as respondents suggested that students in 
grade 6 would also be appropriate respondents for 
this survey component. The other focus groups had 
revealed that this was the age that their parents’ 
awareness of safety issues sharpened. Accordingly, 
the decision was made to include 6th through 12th 
grade students as survey respondents for the 
component on safety and discipline in school. 

The focus groups also confirmed substantial 
differences in the perspectives of parents and 
students about their school, and in the information 
provided by parents and by students for this survey 
component. Youth were more informed about all 
aspects of safety and discipline at school than were 
their parents. Some youth commented that they did 
not like to worry their parents and so kept reports of 
unsafe incidents from them. As a result of this 
finding, the survey component was designed with 
identical wording of most of the questions about 
school safety so that analysts could compare parents’ 
and children’s responses. In addition, the focus 
group discussions suggested that both students and 
their parents judge the safety of their schools relative 
to the safety of their neighborhoods, so items were 
included measuring neighborhood safety that would 
provide a useful context for assessments of safety at 
school. 
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Testing Survey Instruments 

Cognitive laboratory research has also been 
employed extensively in the design of 
instruments for the NHES. For example, it was 
used to test the survey instruments for the Early 
Childhood Education component of the 
NHES:91, and for both the School Readiness and 
the School Safety and Discipline components of 
the NHES:93. The purpose of the research was 
to reduce the incidence of measurement error in 
the NHES by examining whether the participants 
1 ) understood the questionnaire items; 2) had the 
knowledge necessary to answer the questions; 3) 
were able to recall the information when asked; 
and 4) would be willing to answer all the 
questions, especially those that might elicit 
sensitive information. Individual interviews and 
focus groups were included in the cognitive 
laboratory research in each test to take advantage 
of the benefits of the two procedures. Interviews 
allowed the researchers to explore selected issues 
to some depth with one respondent at a time. 
Focus groups were intended to capture range 
more than depth. Guided by open-ended 
research questions contained in the protocol, they 
often produced valuable and unanticipated 
findings due to the interaction that took place 
among respondents. Some interviews used 
concurrent methods, both think-alouds and 
concurrent probing. Other interviews and the 
focus group discussions used retrospective 
probes. 

Testing the NHES:91 Early Childhood 
Questionnaire 

The Early Childhood component was designed to 
obtain information from parents about the care 
and education of their children age 3 through 8. 
It required parents to report on care 
arrangements and schooling over the course of 
several prior years, as well as on current 
arrangements. It also measured activities that 
parents might do with their children at home that 



would promote learning. There was concern about 
the ability of parents to recall details years later, and 
their willingness to report on their children and their 
family activities. 

Two rounds of cognitive laboratory research were 
conducted to test the Early Childhood Education 
questionnaire for the NHES:91. The first round 
tested an initial version of the questionnaire, and the 
subsequent round tested the revised version. Each 
round consisted of six cognitive interviews and two 
focus groups. Prior to the focus groups, each 
participant was administered the interview over the 
telephone. Half of the interviews and one of the 
focus groups in each round were conducted with 
parents of preschool children; the other half of the 
activities were conducted with parents of children in 
kindergarten through grade 3. Respondents were 
recruited through day care providers in the Rockville, 
Maryland area, and through the personal networks of 
Westat employees in Gaithersburg and Frederick, 
Maryland. Each participant was paid an honorarium. 

One of the primary purposes of the cognitive 
laboratory research was to test respondent 
comprehension of the terms used in the survey. The 
Early Childhood Education component of the 
NHES:91 focused on nonparental care arrangements 
and participation in preprimary programs as well as 
primary school programs. It was important to know 
how parents understood nonparental care, defined in 
the interview as a "child care arrangement in which 
care is provided by someone other than you or your 
spouse or partner." Following administration of the 
survey, participants in the first round of cognitive 
laboratory research were asked about their definitions 
of nonparental child care. Specific probes were used 
to determine whether respondents included certain 
arrangements, for example, care by a grandparent 
living in the home or a day camp that lasted for a 
short period in the summer. Other probes explored 
whether a parent distinguished between a nursery 
school and a day care with an educational component 
when asked about preschool. 
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In the initial version of the questionnaire, the 
section asking about child care arrangements 
began with a rather lengthy definition of 
nonparental care that named various types of 
arrangements. The cognitive laboratory research 
revealed that respondents did not attend to the 
details of the definition, and were not always 
reporting care arrangements accurately. 
Following the definition was a general query 
about any kind of nonparental care. This 
research indicated that it is more appropriate to 
ask first about specific types of nonparental care 
or early childhood programs rather than asking a 
general question. The findings suggested that a 
shorter definition would more appropriately cue 
the respondent. Because participants in the 
cognitive laboratory research did not always 
count care by a relative if there was no charge 
for that care, a specification was added to the 
introductory definition indicating that they 
should report any regular, nonparental care given 
by relatives or nonrelatives whether or not there 
was a charge for the service. The second round 
of activities tested the new definition and the 
feasibility of collecting information on each type 
of arrangement separately. The more specific 
questions prompted respondent recall and cued 
respondents as to the types of arrangement in 
which the study was interested. Although this 
involved asking more questions, the cognitive 
burden on the respondents was reduced because 
they did not have to keep multiple criteria in 
mind while answering. 

The cognitive laboratory research also tested 
whether terms that were used in the initial 
questionnaire, such as preschool, nursery school, 
prekindergarten, and day care center, might hold 
different meanings for respondents^ The findings 
indicated that parents did indeed have different 
definitions, particularly for preschool. In the 
original version of the questionnaire, some 
respondents gave information about their child’s 
day care center when asked about day care, and 
then gave the same information again when 
asked about preschool. The subsequent version 



of the questionnaire did not use the term preschool. 
Instead, respondents were asked about day care 
centers, including whether the program had an 
educational component, and then about nursery 
school, prekindergarten, or Head Start programs 
grouped together. When asked about the latter, they 
were instructed to answer only for programs for 
which they had not already provided information in 
order to reduce the incidence of double counting. 

The questionnaire also asked parents to specify the 
type of nonparental care their child had received 
each year between birth and age 5. Participants had 
little difficulty accurately counting the number of 
care arrangements in which their child was currently 
participating. * They were also able to recall past 
arrangements for their child; however, they found it 
difficult to specify the arrangements on a year-by- 
year basis, particularly if they had had several 
arrangements over the years. Participants in the 
cognitive laboratory research explored ways to make 
recall over that period of time easier. They 
discussed whether it was easier to think in terms of 
the ages of their children, months, seasons, or school 
years and summers. However, it appeared from the 
discussions that parents of the older children in the 
survey would not be able to report the number of 
arrangements accurately, even with suggestions that 
might cue them. Consequently, the revised 
questionnaire contained only one question asking 
parents of primary school children whether the child 
had received nonparental care, either home based or 
center based, from birth to age 5. 

Focus group participants were also asked to comment 
on the sensitivity of the items and their willingness 
to answer questions about their children’s care 
arrangements and family activities. They indicated 
a willingness to answer questions on these topics; 
however, it was apparent that parents were inclined 
to answer some questions in a manner that would put 
their parenting in a favorable light, particularly 
questions about activities with their children. Parents 
want to be thought of as competent and caring, and 
they admitted in their discussions that they might 
have a tendency to exaggerate time spent with a 
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child and the types of activities in pvhich they 
engaged by estimating on the high side. For 
example, although the question referred to the 
past week, some parents said that they would 
probably respond about a “typical” week, if the 
past week had not included many activities. 
This finding emphasized the need to both word 
and order questions carefully to minimize 
overreporting. 

Testing the NHES:93 School Readiness and 
School Safety and Discipline Questionnaires 

Cognitive laboratory research also played a 
significant part in testing questionnaires for the 
School Readiness and School Safety and 
Discipline components of the NHES:93. For the 
School Readiness component, nine individual 
interviews and three focus groups containing 
between six and nine participants each, were 
conducted. Approximately one-third of the 
interviews used the think-aloud procedure, one- 
third used concurrent probes, and one-third used 
retrospective probes. Separate focus groups were 
held with parents of children not yet in 
kindergarten, parents of children in kindergarten, 
and parents of children in primary grades. For 
the School Safety and Discipline component, 1 1 
interviews, 5 with parents and 6 with students, 
were completed, and 4 focus groups, 2 with 
parents and 2 with students, were conducted. 
Parent and student focus groups were structured 
by grade of child, either 6th through 8th grade, 
or 9th through 12th grade. Written consent for 
each student’s participation was obtained in 
advance from his or her parent. Parent 
participants and student participants also signed 
consent forms before the discussion began. All 
participants were paid an honorarium. 

Participants were recruited through flyers posted 
in businesses, such as grocery stores and office 
buildings, and in apartment and housing 
complexes with ethnically diverse residents 
located in the Rockville, Maryland, area and 
through the personal networks of Westat 
employees in Frederick and Gaithersburg, 
Maryland. Recruitment criteria included 



diversity on racial/ethnic, gender, and socioeconomic 
status as measured by highest occupation in the 
family. An additional criterion for the School 
Readiness research was having representation of a 
range of participation in types of nonparental care, as 
well as in parental care. For the School Safety and 
Discipline component, limits were placed on the 
number of students or parents of students attending 
private schools who could take part in the research, 
and parents who had ever worked for a school 
system or held office in a parent-teacher association 
were excluded, as were students or parents of 
students attending alternative schools because of 
behavior problems. 

The results of the cognitive laboratory research 
conducted with youth participants to test the School 
Safety and Discipline questionnaire illustrate further 
the utility of including these methods in the design 
of a survey. Particular care was taken with the 
wording of the items for this survey component 
because both parents and students in grades 6 
through 12 would be answering many of the same 
questions. Questions were crafted to appropriately 
cue each population and to be easily comprehensible 
to youth as young as 1 1 years old. Nevertheless, the 
cognitive laboratory research revealed some lack of 
clarity. For example, the test questionnaire contained 
questions about knowledge of and victimization by 
bullying. Youth appeared to have no difficulty 
understanding this question and responded in the 
interviews with correct definitions of bullying. 
However, a more extended discussion that took place 
in the focus group of younger students revealed that 
they had a correct, but narrow, perception. They did 
not think a person was being bullied unless he or she 
was under immediate physical threat. They failed to 
count cases of extortion of lunch money, for 
example, in which the intimidation was pervasive but 
subtle. As a result of this finding, the final version 
of the questionnaire included a definition of bullying 
to cue the respondents to the broader meaning of the 
term. 

Cognitive laboratory research also provided 
reassurance that some terms, thought to be too 
sophisticated perhaps, were likely to be readily 
understood by respondents. For example, youth were 
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asked if they would think that any fight that took 
place between some white students and some 
black students was racially motivated. The 
participants in this research indicated they 
understood the term "racially motivated" as it 
was intended. The students asserted that they 
would know the cause of fights at their schools, 
and only incidents caused by specific animosity 
between racial groups, not just any incident that 
involved both blacks and whites, would be 
labeled racially motivated. 

The individual interviews were particularly 
valuable in assessing respondent comprehension 
and guiding revision of the School Safety and 
Discipline instrument. Youth had difficulty with 
the think-aloud procedure but were able to recall 
their thought processes in response to both 
concurrent and retrospective probes. Often, 
inconsistencies in their responses revealed a lack 
of comprehension, and the ability to explore 
meanings at length with one participant at a time 
was constructive. In some cases, comments 
made only a single time in the research had an 
impact on the redesign of the questionnaire, and 
several items were reworded to improve 
respondent comprehension. For example, 
because one youth interpreted get the following 
things on school property (referring to tobacco, 
alcohol, and other drugs) as carrying them onto 
the school grounds rather than obtaining them on 
school property, the wording was changed to get 
the following things at school or on the school 
grounds. Not only were respondents’ specific 
confusions and suggestions about improving 
clarity helpful, but also the research yielded 
subtle intimations about adolescent, and 
particularly preadolescent, cognitive abilities that 
inspired changes in item wording. For example, 
because a youth thought under the influence of 
alcohol meant under the age to drink legally, 
separate words, not just whole phrases were 
examined for potential confusion for respondents. 
Because of the findings from the interviews, for 
example, the survey asked about students who 
were drunk or showed the effects of alcohol at 
school. 



Findings from both the individual interviews and the 
focus groups revealed that the youth had difficulty 
with questions that required them to think about their 
school experiences in the abstract. For example, they 
had difficulty evaluating whether the school was 
preparing them well for further study or for a future 
job, or whether most of the students at the school 
were well behaved. On the other hand, they had 
little difficulty with questions that asked about 
personal experiences, such as whether they enjoy 
school or if they are challenged by their classes. 
This preference for the personal was reflected in the 
divergent meanings assigned to the term classmate. 
Some thought about the class leaders and some about 
the troublemakers, while others thought of their own 
friends. However, when asked about your friends at 
school, the youth participants described a clearer 
frame of reference. They reported that they knew 
exactly about whom they were thinking. This 
finding led to rewording questions, wherever 
appropriate, to personalize the reference for the 
respondent. For instance, instead of asking about 
pressure from other students at school to use tobacco, 
alcohol, or other drugs, the question asks if the 
youth’s friends at school think it is all right to smoke 
cigarettes, etc. And rather than asking how easy or 
difficult it would be for students to obtain those 
substances at the school, the question was worded to 
ask how easy or hard it would be for the youth 
respondent to obtain the substances if he or she 
wanted to. As a result of the cognitive laboratory 
research, the data obtained from youth are likely to 
be much more accurate. 

Evaluating Respondent 
Comprehension During Data 
Collection Through Taped Interviews 

An additional data quality activity that has been 
conducted in the NHES is recording and analyzing a 
small number of interviews taped during data 
collection. This activity provides another check on 
questionnaire wording in terms of respondent 
comprehension and the ability of respondents to 
provide accurate responses. The research was 
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conducted during the NHES:91 and the 
NHES:93. Different analysis techniques were 
used for each survey. For the NHES:91, content 
analysis of the transcripts of the taped interviews 
was conducted. For the NHES:93, a different 
methodology based on coding interviewer- 
respondent interaction on an item-by-item basis 
was used. 

Content Analysis of Taped Interviews 

During data collection for the NHES:91, a small 
number of extended interviews were recorded 
and subjected to a systematic content analysis, 
although not coded item by item. The analysis 
of the taped interviews took a linguistic and 
holistic approach to the interview process. The 
cumulative effects of the telephone interview 
process, and of the wording of and responses to 
early interview items on respondents’ 
comprehension of later terms were considered. 
The analysis was conducted by staff who were 
not involved in the design of the survey 
instruments and who had not participated in the 
earlier cognitive laboratory research. The 
purpose of the research was to assess respondent 
comprehension during actual telephone 
interviewing. 

Taping was done during the last 2 weeks of data 
collection, and only cases that had never been 
coded a refusal (i.e., to the Screener or to 
another extended interview in the household) 
were eligible. Respondents were asked for 
permission to record their interview after they 
had completed the screening questions, but 
before the extended interview began. If any 
reluctance on the part of the respondent was 
detected, the interview was not taped. Thus, 
participation was voluntary, and this should be 
noted when interpreting results. Twenty-one 
interviews (1 1 for the Early Childhood Education 
component and 10 for the Adult Education 
component) were taped and analyzed. Two 
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additional interviews were taped but not analyzed 
because the extended interview was not completed. 

Examples from the taped Adult Education interviews 
demonstrate the contribution of this research to 
improving data quality in future surveys. In the 
Adult Education survey, the respondents were read a 
definition of adult education activities. Some of the 
taped interviews revealed that respondents recalled 
participation in adult education activities late in the 
interview, including participation in activities that 
should have been included in earlier responses. 
Respondents seemed to hold clear definitions of what 
constituted adult education, but those definitions 
were personal and varied widely. (Had cognitive 
laboratory research been conducted on the Adult 
Education component during the design of the 
NHES:91, this might have been learned earlier.) 
This finding indicated that cognitive research during 
the design of future questionnaires should explore 
ways of cuing respondents more accurately to the 
definition used in the survey, as well as exploring the 
feasibility of cueing them earlier in the interview. 

The issue of sidetracking, in which respondents are 
still considering a previous question or their response 
to a previous question when the next question is 
asked, also emerged from analysis of the taped 
interviews as an important data quality issue. For 
instance, respondents were read a list of types of 
educational activities that included full-time, degree- 
seeking education, part-time, degree-seeking 
activities, and all other types of educational 
activities. These types were intended to cue them to 
their possible participation but were not intended to 
be mutually exclusive or exhaustive. In some of the 
interviews, respondents were still pondering one type 
of activity as the interviewer read another, resulting 
in an increased potential for inaccurate responses. 
This suggested that future design activities should 
test shorter and more distinct cues for the different 
types of activities. Also, interviewer training should 
emphasize allowing respondents ample time to 
absorb the meaning of an item. 
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Interviewer-respondent interaction during the 
interview was also considered. The Adult 
Education respondents were asked questions 
about full-time, degree-seeking education, part- 
time, degree-seeking activities, and all other 
types of educational activities. They were asked 
about the number of activities, names of courses, 
and names of providers. It was often necessary 
for interviewers to clarify for respondents 
whether some activities constituted a separate 
course or activity. For example, one respondent 
was confused about whether private instruction 
taken to help pass a college course was part of 
the course or should be counted as a separate 
activity. The need for interviewer clarification 
pointed to items that could be improved for 
future administrations. Issues such as providing 
more versus providing fewer cues and the 
placement of cues in the interview would be 
appropriate to consider in cognitive laboratory 
activities for future surveys. 

Similarly, the interview required the repetition of 
a series of questions about each course (for up to 
four courses) taken during the past 12 months. 
In some interviews it was quite clear that all 
courses were taken for the same reason, and a 
few of the interviewers mentioned that reason 
first so the respondent did not have to hear the 
same lengthy list repeated again and again. The 
subsequent administration of this component in 
the NHES:95 included methods to reduce the 
repetition in the survey. 

Interviewer-Respondent Behavior Coding 

The analysis strategy used for the NHES:91 
taped interviews led to valuable insights that 
influenced the design of subsequent surveys. 
However, building on recent research on coding 
respondent and interviewer behavior (Oksenberg, 
Cannell, and Kalton 1991), project staff 
employed another way of coding taped interview 
data for the NHES:93. A structured coding 
scheme was developed for each item in the 
survey that would point to areas of the School 
Readiness and School Safety and Discipline 
questionnaires in which deviations from the 
question wording occurred or for which 
interviewer clarification was required. This 



method provided a quantifiable indication of how 
well the questionnaires allowed the interviewers to 
follow prescribed procedures and evoked from the 
respondents an answer that could be coded using the 
established response categories. However, like the 
content analysis conducted in 1991, it did not 
provide reasons for deviations from the questionnaire 
structure. 

Seventy interviews, 25 School Readiness interviews, 
25 School Safety and Discipline parent interviews, 
and 20 School Safety and Discipline youth 
interviews, were taped during data collection and 
analyzed by two specially trained coders. 
Interviewer behavior was rated according to the 
following five codes: interviewer read the question 
exactly as worded, interviewer read the question with 
only a minor wording change, interviewer read the 
question with a major wording change, interviewer 
clarified the question for the respondent, or 
interviewer displayed some affect. Respondent 
behavior was coded using six categories: respondent 
gave a "correct" response (i.e, one that fit a precoded 
response alternative), respondent interrupted the 
interviewer before the question was completed, 
respondent clarified the question, respondent 
qualified the answer with respect to accuracy, 
respondent did not provide an adequate answer, or 
respondent expressed sensitivity to the question. (See 
Appendix C for an example of a coding form). 
Fourteen interviews were coded by both of the two 
coders. The greatest discrepancy was between "exact 
wording" and "minor wording change." For 
example, one coder coded pausing as a minor 
wording change, and the other did not. When the 
"exact wording" and "minor wording change" 
categories were collapsed, overall inter-rater 
reliability was high, ranging from 78 percent to 84 
percent over the three types of interviews. 

The analysis produced the number of times and the 
percentage that each code was used for each of the 
three interview questionnaires. It also permitted 
evaluation of each question according to the codes 
assigned across all of the interviews of that type. 
This allowed identification of specific questions that 
had been subject to major wording changes by the 
interviewer, or that had caused the respondent to ask 
for clarification or to produce a response that was 
not initially codable. 
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Valuable information was obtained from this 
analysis. For example, the introductions at the 
beginning of sections of the questionnaires, 
intended to cue respondents to a change in topic, 
were coded as having the most minor wording 
changes. This may be due to interviewers who 
are responding to the conversational demands of 
the interview process by trying to place the new 
topic in the context of what has gone before. 
The analysis also highlighted the difficulty that 
respondents have with series of questions that 
have a Likert response scale, such as strongly 
agree, agree, disagree, and strongly disagree. 
The information gleaned from the analysis of 
taped interviews will be useful in the design of 
new NHES surveys, both in a general sense, and 
when the specific topics are included in a future 
data collection. 



Summary 

The National Household Education Survey has 
incorporated cognitive laboratory research and 
analysis of taped interviews as integral parts of 
its design process. Individual interviews, using 
concurrent and delayed cognitive laboratory 
methods, and focus groups guided by lengthy 
protocols contributed to the assessment of the 
feasibility of topical components of the survey 
and to testing the questionnaires to reduce 
measurement error. Taping and analysis of live 
interviews measured interviewer and respondent 
behavior during data collection and contributed 
to the understanding of words and items that 
may cause difficulty. The findings will guide 
cognitive laboratory research for future survey 
administrations. The NHES will continue to 
explore new ways to utilize cognitive laboratory 
research and other data quality activities and to 
adapt the work of other researchers to enhance 
coding schemes used in the analysis of the 
NHES data quality. 
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Written consent was obtained from each participant. 



NHES: 93 SCHOOL SAFETY AND DISCIPLINE 
MODERATOR’S TOPIC GUIDE FOR PARENT GROUPS 
I. Warm-up and Explanation (10 minutes)* 

A. Introduction 

1. Thanks for agreeing to come, 

2. Your participation is very important. 

3. A focus group is like an opinion survey, only with very broad, general questions. 

B. Purpose of Focus Group 

1. This evening we will be talking about safety and discipline in the junior high (high 
school). Information from this group and other groups with parents and students will 
be used to help develop a questionnaire on this subject for a survey of parents and 
students across the country. 

2. Everyone has different experiences; all of your ideas, comments and suggestions will 
be helpful. 

3. There are no right or wrong answers. It is all right to disagree with other comments. 
We hope to get many points of view. 

4. All comments - both positive and negative - are welcome. 

C. How the Group Works 

L The session is being tape recorded, and observers are present behind the one-way 
mirror. However, all comments are confidential - used for research purposes only. 

2. This is a group discussion, so you need not wait for me to call on you. Please speak 
one at a time so the tape recorder can pick up everything. 

3. We have a lot of ground to cover, so I may change the subject to move ahead. Please 
stop me if you want to add something. 

D. Introduction of Participants 

1. Let’s take a moment to go around the table and introduce ourselves. Tell us your first 
name and what you like to do in your spare time. Also tell us how many children you 
have and the age and grade of your child in junior high (high school). 



*Participants were asked to sign a consent form in the waiting room. 
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Written consent was obtained from each participant. 



II. School Safety (30 minutes) 

* A. I’d like to start out talking with you about occurrences that may make your child feel afraid 

or unsafe at school, OR may cause you to worry about him or her. These occurrences may 
take place either during the school day or while your child is going to or from school (I’m 
NOT talking about environmental hazards such as asbestos, dangerous playground 
equipment, etc.) . What actions, behaviors, events come to mind? (LIST ON EASEL) 
(Probe: physical/verbal abuse, theft, vandalism, substance use, locker crime) 

* B. How many of you feel these kinds of occurrences at school have been a personal issue for 

your child at any time in junior high (high school) (TAKE COUNT)? In general, what 
aspects of school safety concern or worry you the most? 

* C. Let’s consider something else (something you touched on earlier) - the issue of disruptive 

behavior, whether in the classroom, hallway, playground, school bus, etc. What kind of 
disturbances or infractions contribute to the atmosphere of a school being unsafe or not 
conducive to learning? (ADD ITEMS TO LIST ON EASEL) 

D. Tell me how important you consider these major to minor disturbances that we’ve mentioned 
to be? Why do you say that? 

* 1. What impact does it (can it) have on a child’s behavior, sociability, learning, feelings 

toward school, attendance, emotional state (e.g., fear)? 

(Probe: Does it interfere with learning? What percentage of class time do you feel 
teachers spend dealing with disruptive behavior?) 

2. (Opt. ) Is there another issue you would compare it to, or put it on the same par with? 
How do you see it in relation to other issues parents of junior high/high school 
students have to deal with? 

E. How would you rate your child’s school in terms of it being a safe, pleasant and a 
conducive place to learn? Where would you put it on a scale of 1 - 5 where 1 is "not at 
all safe/pleasant/conducive to learning" and 5 is "very safe/pleasant/conducive to 
learning." (GO AROUND TABLE TO GET RATINGS) 

1 . Those of you who gave higher ratings (say, 4 - 5), how do you know that your 
child’s school is safe and provides a proper atmosphere for learning? 

2. (Opt.) Probe: Where/how do you get your information — e.g., Have there been 
meetings with parents to explain school security and safety? Has the PTA looked 
into the situation, made recommendations and did the administration listen/act? 

Have the local police been visible at the school - talking at assemblies, etc?) 

F. (Opt.) What do you actually look for to determine if a school is safe? In other words, 
how do you evaluate a school -- what criteria do you use? (Probe: Is it realistic to expect 
a school to be 100 percent safe?) 



♦High priority question. 
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Written consent was obtained from each participant. 



* G. What would you do (or have you done) if you thought your child’s school was unsafe? 

(Probe: switch to another school — public or private; restrict your child’s behavior; band 
together with other parents; speak to school officials) 

H. How do you know when a school is unsafe? (Why did some of you assign ratings of 
1 - 2 to your child’s school?) Can you give me some examples of what makes a school 
unsafe? 

1 . What role does rumor play? How would you react if you heard that someone at 
your child’s school had been attacked? (Probe: Does that make the school unsafe?) 

2. (Opt.) Some people are afraid that safety in the schools may be getting worse. 

What does "getting worse" mean to you? 

III. Alcohol and Drug Use (25 minutes) 

* A. Teenage alcohol and drug use has already come up in our discussion. At this point, I’d 

like to talk more about these issues. Before we dp, however, I want to pass out a piece 
of paper with some statements on it, and I’d like you to indicate on a scale of 
1 - 5 how much you agree (or disagree) with each one. Please be as honest as you can; 
do not put your names on these sheets. 

(DISTRIBUTE HANDOUT CONTAINING THE FOLLOWING STATEMENTS AND A 
: 5-POINT AGREE/DISAGREE SCALE) 

1. Young people should not be allowed to drink alcohol until they reach the age of 
18. 

2. Drinking is taboo by anyone in my (our) household. 

3. I think some parents get overly upset by their child using marijuana (grass) on an 
occasional basis. 

4. I do not allow my child to attend parties or other social events unless I check to be 
sure an adult chaperon will be present. 

5. I do not allow my child to have parties at my house with alcohol (including beer, 
wine, wine coolers). 

6. Experimenting with alcohol or marijuana is just a part of growing up today. 

7. Smoking is a relatively harmless form of teenage rebellion. 

8. If my child had to experiment with something, I would rather it were alcohol than 
marijuana. 

9. I feel I am less tolerant of teenage drinking than other parents of teenagers I know. 

10. I could understand if my child tried cocaine or heroin only once. 
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Written consent was obtained from each participant. 



(HAVE PARTICIPANTS PASS PAPERS FORWARD) 

B. How common do you think it is for junior high (high school) students to attend functions 
outside of school where alcohol is available? How about marijuana? Hard drugs (cocaine, 
heroin)? Amphetamines? 

C. Let me get your reaction to something else. A parent in one of my groups said recently 
that he and his wife occasionally offer their teenagers a drink so that having a drink will 
not be a big deal to them - in other words, it won’t be an allurement or "forbidden 
fruit." How do you feel about this? 

D. Some statistics have shown that the majority of teenagers try alcohol in high school. At 
what age would you not be alarmed if you heard your son or daughter were at a party 
and had a few beers or wine coolers? Conversely, at what age (or under what conditions) 
would you be alarmed to hear this? 

E. Do you think you would actually know if your child were using cigarettes, alcohol, 
marijuana, or other drugs? How would you know? (Probe: Have you ever talked with 
your child about his/her use of cigarettes, alcohol, or drugs, or do you just know?) 

F. Do you have any rules in your household about drinking? (What are they?) How about 
marijuana use? Other drugs? Smoking? Are there any consequences or punishments for 
using these things? (What are they?) 



IV. Discipline Policy (Rules and Regulations) (35 minutes) 

« 

A. Now I’d like to change the subject somewhat and talk to you about discipline policy. 
Before asking you some general questions; however. I’d be interested in hearing what 
would happen at your child’s school if the following situations arose... 

1. A student at your child’s school was caught drinking alcohol during the school 
day? (Probe: Would this be a fair action if your child had just a few sips of beer 
that another student had brought to a school football game?) 

2. One student physically attacked another student? (ProbeN Would this be a fair action 
if your child had gone through several weeks of verbal bullying and either your son 
got into a fistfight, or your daughter slapped the person bullying her?) 

3. (Opt.) A student shouted obscenities at a teacher who was reprimanding him/her? 
(Probe: Would this be a fair action if your child muttered the same obscenities 
while walking away after a reprimand?) 

B. Do you feel that discipline (or the lack thereof) is a problem in the junior high (high) 
schools? When I use the term "school discipline," I’m talking about disruptive behavior 
in the classroom as well as some of the other items we discussed earlier. (We’re not 
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Written consent was obtained from each participant. 



talking about discipline for dress code infractions or tardiness to school.) Why do you say 
that? 



C. What do you think school officials should be doing to ensure safe and well-disciplined 
schools? 



* 




1. Should suspensions and expulsions be used as forms of punishment? 

2. The mayor of Washington has suggested introducing corporal punishment back into 
the schools — what do you think of that? 

3. How do you feel about random locker searches, metal detectors, drug-sniffing dogs, 
random urine testing, hall patrols? 

D. Do you know whether or not your child’s school has a discipline policy? 

1 . Do you know if it is written down? What form does it take (e.g., handbook)? 

2. Have you ever seen/read it? Do you have a copy? 

3. How was it developed? Who put it together? (Probe: Was there any input from 
parents? Students?) 

4. What does it cover (e.g., disruptive behavior, substance use)? 

E. How do you feel about the discipline policy at your child’s school? Is there anything you 

particularly like about it? Particularly dislike about it? 

1 . Do you consider it too strict? Too lenient? In what way(s)? How do you judge 
this (i.e., what standard are you using)? 

2. Do you feel that the school acts (i.e., sets policy) only after an incident occurs or is 
it more proactive in nature, setting policy in advance of something happening? 

F. Having a policy is one thing; how it functions can be something else. What do you think 

about how the discipline policy at your child’s school actually works? 

1. Are regulations consistently and fairly applied? What standard are you using to 
evaluate this? 

2. Are there separate rules (or are the rules interpreted differently) for first-time 
offenders versus repeat offenders? 

3. Is the punishment the same for possessing drugs, using drugs, and distributing 
drugs (i.e., encouraging others to use)? How do you feel about this? 

4. Do you know if there is a way for the discipline policy at the school to get changed 
or amended? (Probe: Can you think of any circumstances that might cause you or 
other parents to appeal for a change in how the policy is applied?) 
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* G. We’ve talked so far about safety and discipline at the junior high (high school) level. I’d 

like you to think back for a minute to the time when your child was in elementary/grade 
school. Would you say that safety and discipline were of greater concern, lesser concern, 
or of the same concern to you then as they are now? (TAKE COUNT) 

1. What safety and discipline concerns did you have for your child in elementary/grade 
school? (What things made your child feel unsafe or afraid, or caused you to worry?) 

2. At what age/grade level did you become concerned about the possibility of disruptive 
behavior and/or substance use in the school and its possible impact on your child? 

V. Alcohol Drug Education in the School (20 minutes) 

A. In the time remaining, I’d like to get back to the subject of alcohol and drugs and talk with 
you about alcohol and drug education in the junior high (high school). Is it important for 
you to have alcohol/drug education as part of the school curriculum? 

1. (Opt.) Probe: What aspects of your child’s education do you consider to be more 
important? Less important? On a par with alcohol/drug education? 

* B. Does your child’s school currently have an alcohol/drug education program? (TAKE 

COUNT) 

1 . What do you know about it? 

2. Are drug and alcohol education handled together (as part of same course) or 
separately? 

3. Do you know what overall message your child is getting? (Probe: Is it a "no use" 
message or a "responsible use" message?) Does this agree or conflict with your own 
attitude toward education on this subject? 

C. How do you feel about the adequacy of the alcohol/drug education your child is given in 
school? How are you making this judgment? 

D. (Opt.) Are there any drug education programs for parents at your child’s school (e.g., to 
help them form their own peer or support group)? 

VI. Wrap-up 

A. We’ve come to the end of the session. Is there anything you’d like to add on any of our 
topics tonight? 

B. Thank you for coming. Your opinions have been very valuable to us. Please see 
receptionist on your way out. 
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Moderator’s Topic Guide for Adolescent Groups 




27 



Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 

NHES: 93 SCHOOL SAFETY AND DISCIPLINE 



MODERATOR’S TOPIC GUIDE FOR ADOLESCENT GROUPS 



Warm-up and Explanation (15 minutes) 

A. Introduction 

1. Thanks for agreeing to come. 

2. Your participation is very important. 

3. A focus group is like an opinion survey, only it’s done in a group. 

B. Purpose of Focus Group 

1 . This afternoon we will be talking about safety and discipline in the junior high (high 
school). Information from this group and other groups with students like yourselves 
will be used to help develop a questionnaire on this subject for a survey of parents and 
students across the country. 

2. Everyone has different experiences; all of your ideas, comments, and suggestions will 
be helpful. 

3. There are no right or wrong answers. It is all right to disagree with other comments. 
We hope to get many points of view. 

4. All comments - both positive and negative - are welcome. 

5. I would like to hear from everybody. 

C. How the Group Works 

1 . The session is being tape recorded and observers are sitting behind the one-way mirror. 
However, all comments are confidential - used for research purposes only. 

2. This is a group discussion. You need not wait for me to call on you, but please speak 
one at a time so the tape will be clear. Also, be sure to give others in the room a 
chance to speak. 

3. We have a lot of things to talk about, so I may change the subject in order to move 
ahead. 



Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 

4. Before we start, I have consent forms for you to sign. I’ll read it out loud and then 
you can sign it and pass it to me. [NOTE: Prior written consent was obtained from 
each student’s parent.] 



D. Introduction of Participants 

1 . I’d like to take a few minutes for us to get to know each other. Instead of having you 
introduce yourselves (the way it’s usually done), I’m going to ask you to introduce the 
person sitting next to you. Take a couple of minutes right now to talk to the person 
on your right. Find out his/her first name, age, grade in school, name of school 
attending, number of brothers and sisters and what hobbies or after-school activities 
the person is into (i.e., what he/she likes to do in his/her spare time.) (LIST ABOVE 
ITEMS ON EASEL FOR REFERENCE) 

II. School Safety (30 minutes) 

* A. I’d like to start out talking about things that make you (or your friends) feel afraid or unsafe 

at school. These are things that may take place either while you are in school or while you 
are going to or from school. What comes to mind first? What different things can you think 
of? (LIST ON EASEL) (Probe: physical/verbal abuse, theft, vandalism, substance use, locker 
crime) 

* B. Which things on this list concern you and your friends the most? Write the word "Most" on 

your pad and next to it the item on the easel that concerns or worries you most at your 
school. Which one worries you the next most after that? Write "Next Most" on the next line 
and then the item from your experience that fits best. 

* C. (IF NOT ALREADY MENTIONED) Let me bring up something else here — disruptive 

behavior. Does this go on at your school? What do you think of when I say "disruptive 
behavior?" (What kinds of disruptive behaviors make the atmosphere at school seem unsafe 
or make it difficult for students to learn?) (ADD TO LIST) 

1. Do you think teachers spend too much or too little time dealing with disruptive 
behavior? 

* D. Are these various things we’ve put up on the easel important to you? Why do you say 

that? 



1 . Do you think things are different in a safe school than in an unsafe school? (Probe: 
How easy or hard is it to learn in a school that doesn’t have a safe atmosphere?) 



*High priority question. 



Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 



2. How else are students affected (e.g., in terms of behavior, sociability, feelings toward 
school, attendance, fear)? 

E. (Opt.) How do you know if a school is safe (any school, not necessarily your school)? Write 
down three things that would tell you a school is safe. 

F. Think of someone you know who goes to an unsafe school and write down three things 
that would tell you a school is unsafe . 

1. How would you feel if you heard a rumor that someone had been attacked at school? 
(Probe: Does that make you think the school is unsafe?) 

G. How safe do you think your school is? Where would you put your school on a scale of 1 - 

5 where 1 is "very unsafe/a difficult atmosphere to learn in" and 5 is "very safe/an easy 
atmosphere to learn in?" (HAVE SCALE ON EASEL) Write down the number on the scale 
that tells how safe you think your school is. 

1. Can you give me an example of something that happened at your school that made it 
seem unsafe; either something that happened to you, or to one of your friends. How 
did this make you feel? 

2. How do you know or find out about other students who are victims of crimes? 

H. What do students do if they think their school is unsafe? Do they change their behavior or 
habits in any way to protect themselves (e.g., avoid certain corridors or bathrooms at 
school)? 



III. Alcohol and Drug Use (20 minutes) 

A. Use of alcohol and drugs has come up a number of times in our discussion. I’d like to talk 
with you more about this subject, but before we get into that, I want to pass out a piece of 
paper with some statements on it, and I’d like you to indicate on a scale of 1 - 5 how much 
you agree (or disagree) with each one. Please be as honest as you can; there is no need to 
put your names on these sheets. 

(DISTRIBUTE HANDOUT CONTAINING THE FOLLOWING STATEMENTS AND A 
5-POINT AGREE/DISAGREE SCALE. BE SURE EVERYONE UNDERSTANDS 
SCALE). Alcohol means beer, wine, wine coolers and hard liquor. 

1. Young people should not be allowed to drink alcohol until they reach the age of 18. 

2. Drinking by teens is not allowed in my house. 

3. I think some parents get overly upset by their child using marijuana (grass) on an 
occasional basis. 
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Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 

4. I cannot go to parties or other social events unless my parents know, an adult chaperon 

will be there. "1 

5. I cannot have parties at my house with any alcohol. 

6. Trying alcohol or marijuana is just a part of growing up today. 

7. Smoking cigarettes is a pretty harmless way for teenagers to show their independence. 

8. Lots of teenagers try drugs like coke, heroin, or amphetamines once or twice. 

9. It is easy for kids I know at my school to obtain alcohol. 

10. It is easy for kids I know at my school to obtain marijuana. 

11. It is easy for kids I know at my school to obtain other drugs such as coke, heroin or 

amphetamines. 

(HAVE PARTICIPANTS PASS PAPERS FORWARD) 

B. Do you remember what grade you were in when your classmates were first thinking 
about smoking cigarettes? How about drinking alcohol? Using marijuana? Other drugs? 

C. Do you think your parents would know if you were using cigarettes, alcohol, marijuana, 
other drugs? How would they know? 

D. Are there any rules in your household about drinking? (What are they?) How about 
cigarettes? Marijuana? Other drugs (coke, heroin, amphetamines)? Are there any 
consequences or punishments for using these things? 

(What are they?) 

1 . Have your parents ever talked to you about using cigarettes, alcohol, marijuana, other 

drugs? How do you feel about their views on these subjects? 

— • \\ 

2. At what age do you think it is all right to have alcohol at a party? (At what age, or 
under what conditions, do you think this would be wrong?) 

Discipline Policy (Rules and RegulationsH 40 minutes) 

* 

A. Now I’d like to change the subject somewhat and talk to you about discipline policy at 
school. What does the word "discipline policy" mean to you? What else do you call it? 
(AFTER PARTICIPANTS GIVE THEIR INTERPRETATION) In this part of the discussion, 
when I use the term "discipline policy," I will be referring to the rules and regulations at 
school that cover disruptive behavior in class as well as some of the other behaviors we 
listed on the easel. (We’re not talking about discipline for violating the dress code or coming 
to school late.) 



Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 

B. Before I ask you some general questions on school discipline, I’d be curious to know what 
would happen at your school if the following situations came up... 

1. A student was caught drinking alcohol during the school day? (Probe: Would this be 
a fair action if your best friend had just a few sips of beer that another student had 
brought to a school football game?) 

2. One student physically attacked another student? (Probe: Would this be a fair action 
if, for example, someone at school was spreading false rumors about you and these 
rumors were hurting your relationships with your friends, so during a confrontation 
with the person, you hit him/her? 

3. (Opt.) A student shouted a couple of four-letter words at a teacher who was chewing 
him/her out? (Probe: Would this be a fair action if you said the same words while 
walking away from a teacher after he/she had yelled at you mostly because he/she was 
being unfair or was in a bad mood, not because you had done something really wrong? 

C. What do you think the school administration should be doing about safety and discipline in 
the junior high (high school)? 

1. Do you think that suspensions and expulsions should be used to discipline students? 

2. Do you know what "corporal punishment" is? You may have heard that the mayor 
of Washington, \DC has suggested that corporal punishment be put back in the 
schools — what do you think of that? 

3. How do you feel about random locker searches, metal detectors, drug-sniffing dogs, 
random urine testing, hall patrols? 

D. Do you know if your school has a discipline policy - or a set of rules or regulations 
about discipline? 

1. Is it written down anywhere? Where (e.g., handbook)? 

2. Have you ever seen or read it? Do you have a copy? 

3. Who put it together? (Probe: Did students have any say in it? How about parents?) 

4. What things does it cover (e.g., disruptive behavior, substance use)? 

E. How do you feel about the discipline policy at your school? Is there anything you like 
about it? Dislike about it? 

1 . Do you consider it too strict? Too lenient? In what way(s)? How do you judge it 
(i.e., is there a standard of some kind you’re using)? 




32 



37 



Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 

F. What do you think about how the discipline policy at your school actually works? 

1. Are rules and regulations applied fairly to different students? What standard are you 
using to say this (how do you know)? 

2. Are there different rules for kids who do something wrong the firsUtime and kids 
who have done it several times? 

3. Is the punishment the same for using drugs, having drugs in your possession, or for 
distributing drugs (i.e., encouraging others to use)? How do you feel about that? 

4. Does the discipline policy at your school ever get changed? What makes this happen? 
Do students ever appeal for a change in the policy? (Under what circumstances?) 

5. Do you feel that the school makes rules only after an incident occurs, or does it set 
policy before something happens? 



Alcohol/Drug Education in the School (15 minutes) 

A. In the time we have left. I’d like to get back to the subject of alcohol and drugs, and talk 
with you a little bit about alcohol and drug education in the junior high (high school). Does 
your school have an alcohol/drug education program? (TAKE COUNT) 

1. Tell me about the program. What is it like? When do you have it? 

2. Are there separate units on alcohol and drugs, or are they all together in the same 
program? 

B. How do you feel about the alcohol/drug education program at your school? Is it good/not 
good? How do you judge it? 

1 . Do you remember anything from it? 

2. Do you think it has affected you? How about your friends? 

3. What overall message are you getting from the program? Would you say it’s a "Don’t 
Use [at all]" message or a "Use Responsibly" message? 

C. Do you think it’s important to have alcohol/drug education as part of the school curriculum? 
Tell me why you say that. 

Wrap-up 

A. We’ve come to the end of the group. (IF TIME) Is there anything you’d like to add on 
anything we’ve talked about? 



Advance written consent was obtained from each student’s parent. 

Written consent was also obtained from each student. 

B. Thanks for coming. Your thoughts have been very helpful to us. Please see the 
receptionist on your way out. 
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Appendix C: 



Form for Behavior Coding of Interviewer-Respondent Interactions from 

Taped Interviews 
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